Blob storage: Blob state machine with a pluggable gRPC data plane - #109
Blob storage: Blob state machine with a pluggable gRPC data plane#109rjhuijsman wants to merge 21 commits into
Blob state machine with a pluggable gRPC data plane#109Conversation
Current Aviator status
This pull request is currently open (not queued). How to mergeTo merge this PR, comment
See the real-time status of this PR on the
Aviator webapp.
Use the Aviator Chrome Extension
to see the status of your PR within GitHub.
|
Before this change, an application's custom HTTP routes (`application.http`) could only be registered for `GET`, `POST`, and `OPTIONS`; the docs listed the `PUT`/`DELETE`/... gap as a known limitation. This blocked serving a plain-HTTP upload endpoint, where `PUT` is the natural verb. Add `application.http.put(...)`, a sibling of the existing `post(...)` that forwards `methods=["PUT"]` to the underlying FastAPI route (the route-capture machinery already supports arbitrary methods; only the public sugar was missing). Update the custom-HTTP-routes documentation to list `PUT` among the supported methods. Co-Authored-By: Claude Fable 5 <[email protected]>
Before this change, `Application.run()` invoked each library's `pre_run(application)` hook, but the `Reboot` in-process test harness (`reboot.aio.tests`) did not. A library that performs application setup in `pre_run` — for example, registering custom HTTP routes — therefore behaved differently under test than in a real run, and its routes were simply absent when brought up via the harness. Call `library.pre_run(...)` for every library in `Reboot.up()`, before deciding whether a local Envoy is needed, mirroring what `Application.run()` does. Libraries must already tolerate being `pre_run` more than once (a test may bring the same application up again after a `down`). This harness behavior is covered by unit tests introduced in a later commit (`reboot/std: add a `Blob` state machine with a gRPC blob data plane`): `blobs_tests.py` brings an application up with the `BlobsLibrary`, whose `pre_run` hook connects to the blob data plane and registers the byte-proxying HTTP routes the tests then exercise — which succeeds only when the harness has invoked `pre_run`. Co-Authored-By: Claude Fable 5 <[email protected]>
Reboot had no first-class way to store large binary objects: state machines hold protobuf state, which is unsuited to multi-megabyte payloads, so applications had nowhere to put user uploads like images or videos. Add `rbt.std.blobs.v1.Blob`, a state machine that is the *control plane* for one immutable-once-committed binary object. Its state holds only metadata — content type, expected/maximum size, upload progress, lifecycle status — while the bytes live in a *data plane* and travel directly between the client and that data plane via URLs minted per part. Uploads are resumable (parts are idempotent by number), sizes are enforced against the real bytes at commit time, and blobs that are never committed expire automatically. The data plane is a gRPC service, `BlobDataPlane` (`data_plane.proto`), deliberately free of Reboot options so that anything can implement it; the control plane discovers it via `REBOOT_BLOB_DATA_PLANE_URL` and calls it to provision uploads, mint URLs, finalize objects, and delete bytes. Keeping the implementation behind a bare gRPC URL means developers can write their own data planes to fit any environment. A data plane whose URLs are not directly reachable by clients (e.g. when run adjacent to an `rbt dev run` that's used over an `ngrok` tunnel) asks, via its `Configuration`, for certain requests to the app's HTTP server (under the reserved `/__/reboot/blob/`) to be forwarded to it; the `Blob` library then registers reverse-proxying routes on the application, so a single application origin serves both control plane and bytes and no second port needs exposing. A data plane whose URLs _are_ directly reachable (e.g. presigned S3) asks for nothing and is never in the application's path. This commit contains just one implementation of a data plane: a filesystem-based server. It binds loopback only and relies on the forwarded-path proxying above, so it works anywhere a Reboot app may be deployed. The `reboot.aio.tests.Reboot` test harness runs the filesystem data plane in-process for every test, so applications using `reboot.std.blobs` work in unit tests out of the box — which is also how this commit is tested. A later commit has `rbt dev run`/`rbt serve run` provide the same data plane for local runs. Authorization model: blob creation is application-mediated (the application enforces quota and size policy), after which the blob's framework-generated random ID acts as a capability (only callers who know the ID can call, and thus read or write, the blob). In addition the backend may limit the identities of callers by setting `uploader_id` and `downloader_ids` at create-time. Co-Authored-By: Claude Fable 5 <[email protected]>
With the blob data plane behind a gRPC interface, an application that uses `reboot.std.blobs` needs a data-plane service to talk to — but local development must keep working out of the box, with no external service to configure. Have `rbt dev run` and `rbt serve run` start the open-source filesystem data-plane server as a background subprocess whenever `REBOOT_BLOB_DATA_PLANE_URL` is not already set (in Reboot Cloud, or via `--env`, it is — and then nothing is spawned), and point the application at it on localhost. The server picks its own ports and reports them through a ready file only once both its gRPC and HTTP endpoints are listening, so there is no port-allocation race and the application can never observe a half-started data plane. Blob bytes live under the application's state directory, so `rbt dev expunge` removes them along with the rest of the state. Co-Authored-By: Claude Fable 5 <[email protected]>
Uploading a file to a `Blob` from the browser means driving the whole multipart protocol — fetching upload instructions, `PUT`ting each part to its URL, reporting ETags, committing, and polling for the result — plus resuming after a dropped connection. Before this change an application author had to write all of that by hand against the generated client. Add `@reboot-dev/reboot-std-react/blobs`: - `useBlobUpload()` — the dead-simple case: given a blob id (from an application RPC, since creation is app-mediated) and a `File`, it uploads every part directly to the data plane, resumes already- confirmed parts, reports progress, and commits. - `BlobUploader` — the same machinery for bytes that don't come from a `File` (media recorders, transforms), with explicit `putPart`/`commit` and a `writable()` stream. - `useBlobDownloadUrl()` — resolves to a URL for a committed blob (e.g. for an `<img src>`), plus a re-exported reactive `useBlob` so any participant — not just the uploader — can render live progress. The part-`PUT` response carries the part's ETag, which the browser must read to report it back; expose the `etag` response header through Envoy's CORS configuration so cross-origin uploads (including direct-to-S3 uploads on the Cloud) can see it. Co-Authored-By: Claude Fable 5 <[email protected]>
The chat-room example only sent text, so it demonstrated nothing about storing binary data. Give it file attachments, wired the way a real app would: message-first, with attachment bytes uploaded after the message is already visible. `Send` now takes a list of attachment descriptors; the servicer checks each against a per-attachment size limit, creates a `Blob` per attachment, embeds the blob ids in the immediately-published message, and returns them. The web frontend then uploads each file into its blob via `useBlobUpload`, and every participant renders the attachment off its reactive blob status — a progress bar while it uploads (visible to everyone, since progress lives on the blob's state), the image once committed. No end-user auth here, so blobs are created with an empty `owner_id` (anyone in the room may upload). The documentation snippets that embed the chat-room proto are regenerated to match. Co-Authored-By: Claude Fable 5 <[email protected]>
1574235 to
6f904f0
Compare
| // Upper bound on the total size in bytes, enforced as parts are | ||
| // reported and at `Commit`. | ||
| optional uint64 max_size = 4; |
There was a problem hiding this comment.
Mutually exclusive with size? Therefore possibly a oneof?
There was a problem hiding this comment.
They're deliberately independent rather than mutually exclusive. CreateRequest.max_size already documents that both may be set, and the implementation enforces both: size exactly at Commit, max_size as a running ceiling on every PartUploaded. A oneof would rule out the useful "I know the exact size, and also never exceed this ceiling" combination.
The Blob field comments didn't say any of that, which is probably what made it look exclusive — they now state the independence explicitly.
Leaving this thread unresolved in case you'd still prefer to restrict it.
There was a problem hiding this comment.
If we know size, then when is it useful to have a max_size that's not exactly size?
- If
max_sizeis smaller: bug, upload can't complete. - If
max_sizeis bigger: allows uploads to grow bigger than what will be accepted at commit.
So if size is set, it seems like we should also use that value as our max_size, and we shouldn't permit max_size to also be set.
There was a problem hiding this comment.
You're right, and my earlier reasoning didn't hold up: if size is known, the ceiling should simply be size. There's no combination where a separate max_size adds anything — smaller is unsatisfiable, bigger just lets bytes accumulate that Commit will reject anyway.
Made both changes:
Schema — size and max_size are now a oneof size_limit in CreateRequest, Blob and InfoResponse, so setting both is no longer expressible rather than merely discouraged.
Enforcement — a declared size now acts as its own ceiling. Previously only max_size bounded the running total on PartUploaded, so a size-only blob would accept unbounded bytes and refuse them at Commit. A single _size_ceiling() helper returns size, else max_size, else None, and is used on PartUploaded, at Commit, and for the ceiling handed to the data plane's CompleteUpload.
Added a case to test_size_validation covering the behaviour that was missing: a part overshooting a declared size is now rejected as it's reported, instead of being stored and refused later. //tests/reboot/std/blob/v1:blob_tests_py passes.
Fixed in 5f06d89.
Before this change, Reboot had no first-class way to store large binary objects: state machines hold protobuf state, which is unsuited to multi-megabyte payloads, so applications had nowhere to put user uploads like images or videos.
This PR introduces the
Blobstate type as the control plane for one immutable-once-committed binary object: its state holds only metadata (content type, size, upload progress, lifecycle, authorization) while the bytes travel directly between the client and a data plane. Data is uploaded to the data plane in parts, each viaPUTto a URL minted by the control plane - compatible with S3 as a data plane, if we ever want that. Uploads are resumable, sizes are enforced against the real bytes at commit time, and never-committed blobs expire automatically.The data plane is a plain gRPC service,
BlobDataPlane(data_plane.proto— no Reboot options, implementable by anything), discovered viaREBOOT_BLOB_DATA_PLANE_URL. Locally (rbt dev run/rbt serve run/ unit tests) a filesystem-based data plane server is run out-of-the-box.Authorization:
Blobhas two mechanisms that combine:Blobcreation is always application-mediated; it can't be done directly from a frontend, creation must go through a backend call. The backend can then use whatever existing auth mechanism they'd like. Assuming theBlobis given a random ID, knowing that state ID acts as a capability: to upload or download you must first know the ID.Blobs can be created with anuploader_idanddownloader_ids; if given these will limit uploads and downloads to only users whose IDs are in those lists.Reviewer hint: review commits in-order.
TESTED: with new unit tests, and by manually running
rbt dev run.